Patient Treatment Resource Utilization Predictor

ABSTRACT

A computer implemented method includes obtaining a schema defining variables representing patient treatment experiences for one or more health conditions, obtaining data from historical patient treatment experiences, the data being arranged in accordance with the schema, training a machine learning system based on a training set of the data for a specified time-period, and generating a model from the trained machine learning system, the model being based on variables selected by the trained machine learning system to predict patient treatment experiences for the one or more health conditions.

BACKGROUND

Managing a health care facility (or group of facilities—enterprise) suchas a hospital can be a very complex task. Patients with various ailmentsthat are admitted to the hospital may utilize different amounts ofresources, such as length of stays, for treatment. Patients with thesame ailments may also utilize different amounts of resources.Complications may increase the utilization of resources. Changes in theway patients are treated may be made in attempts to improve resourceutilization. The results of such changes can be difficult to observe. Itcan also be difficult to compare the performance of various hospitalssince no two hospitals are the same and may have very differentprocesses, staff, equipment, patient demographics, and a multitude ofother factors that may be different.

SUMMARY

A computer implemented method includes obtaining a schema definingvariables representing patient treatment experiences for one or morehealth conditions, obtaining data from historical patient treatmentexperiences, the data being arranged in accordance with the schema,training a machine learning system based on a training set of the datafor a specified time-period, and generating a model from the trainedmachine learning system, the model being based on variables selected bythe trained machine learning system to predict patient treatmentexperiences for the one or more health conditions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a database platform that utilizes a trainedexpert system to generate efficient decision trees to measure resourceutilization according to an example embodiment.

FIG. 2 is a flowchart illustrating a computer implemented method ofpredicting resource utilization in the treatment of patients accordingto an example embodiment.

FIG. 3 is a flowchart illustrating a method of evaluating theeffectiveness of changes made to procedures in the healthcare facilityaccording to an example embodiment.

FIG. 4 is a flowchart illustrating a method of generating a furtherimproved decision tree according to an example embodiment.

FIG. 5 is a block diagram of an example of an environment including asystem for neural network training according to an example embodiment.

FIG. 6 is a block schematic diagram of a computer system to implementdevice and methods for implementing the database platform according toan example embodiment.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings that form a part hereof, and in which is shown by way ofillustration specific embodiments which may be practiced. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice the invention, and it is to be understood thatother embodiments may be utilized and that structural, logical andelectrical changes may be made without departing from the scope of thepresent invention. The following description of example embodiments is,therefore, not to be taken in a limited sense, and the scope of thepresent invention is defined by the appended claims.

The functions or algorithms described herein may be implemented insoftware in one embodiment. The software may consist of computerexecutable instructions stored on computer readable media or computerreadable storage device such as one or more non-transitory memories orother type of hardware-based storage devices, either local or networked.Further, such functions correspond to modules, which may be software,hardware, firmware or any combination thereof. Multiple functions may beperformed in one or more modules as desired, and the embodimentsdescribed are merely examples. The software may be executed on a digitalsignal processor, ASIC, microprocessor, or other type of processoroperating on a computer system, such as a personal computer, server orother computer system, turning such computer system into a specificallyprogrammed machine.

The functionality can be configured to perform an operation using, forinstance, software, hardware, firmware, or the like. For example, thephrase “configured to” can refer to a logic circuit structure of ahardware element that is to implement the associated functionality. Thephrase “configured to” can also refer to a logic circuit structure of ahardware element that is to implement the coding design of associatedfunctionality of firmware or software. The term “module” refers to astructural element that can be implemented using any suitable hardware(e.g., a processor, among others), software (e.g., an application, amongothers), firmware, or any combination of hardware, software, andfirmware. The term, “logic” encompasses any functionality for performinga task. For instance, each operation illustrated in the flowchartscorresponds to logic for performing that operation. An operation can beperformed using, software, hardware, firmware, or the like. The terms,“component,” “system,” and the like may refer to computer-relatedentities, hardware, and software in execution, firmware, or combinationthereof. A component may be a process running on a processor, an object,an executable, a program, a function, a subroutine, a computer, or acombination of software and hardware. The term, “processor,” may referto a hardware component, such as a processing unit of a computer system.

Furthermore, the claimed subject matter may be implemented as a method,apparatus, or article of manufacture using standard programming andengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computing device to implement thedisclosed subject matter. The term, “article of manufacture,” as usedherein is intended to encompass a computer program accessible from anycomputer-readable storage device or media. Computer-readable storagemedia can include, but are not limited to, magnetic storage devices,e.g., hard disk, floppy disk, magnetic strips, optical disk, compactdisk (CD), digital versatile disk (DVD), smart cards, flash memorydevices, among others. In contrast, computer-readable media, i.e., notstorage media, may additionally include communication media such astransmission media for wireless signals and the like.

FIG. 1 is a block diagram of a database platform 100 that utilizes atrained expert system 110 to generate decision trees 115 correspondingto a set of rules for a health care facility, such as a hospital, thatcan classify if a patient is out of norm in terms of resourceutilization. The platform 100 may also be used to predict if a patientwill go out of norm and require more or few resources to ensure propertreatment. The database utilizes a defined schema for training andtesting data to provide accurate predictions while optimizing platformcomputing resources to measure health care facility resourceutilization.

In one embodiment, the predictions correspond to a length of staymetric. In further embodiments many other metrics may be predicted, suchas readmittance following a hospital stay, outpatient visits turninginto a hospitalization, emergency room visits frequency, complicationrate, expenditure, population emergency room visit rate, populationhospital admission rate, etc. Each of these metrics correlate toconsumption of health care facility resources, as well as patient careand experience. For instance, hospital readmission rates may be anindicator of quality of care. A readmission may result from incompletetreatment or poor care of the underlying problem or may reflect poorcoordination of services at the time of discharge or afterwards.Readmissions are not only important indicators of quality, but alsorequire additional resources to handle. Complications can also consumeadditional resources, including increasing length of stay for patients.Some complications are the result of a secondary diagnoses that may havebeen present on admission which affects the risk for complications. Someprimary diagnoses are indicative of the risk for correspondingcomplications.

By classifying or predicting patient resource utilization, patients maybe segmented by high vs. low utilization with the rules. Suchsegmentation facilitates identification of contributing common factorsfor patients to then predict how other people/populations will consumeresources and costs. Knowledge of the contributing factors enables theprovision of input to a health care provider, such as physician or nurseabout a patient before they treat the patient so such providers areready to deal with issues corresponding to the common factors identifiedbefore such issues manifest. The predictions may also identify likelyissues that may arise during treatment to help a healthcare provider tobe prepared for such issues.

The platform may utilize at least two input files. A database schemafile 120 defines the data schema for the lines of input data and a datafile 125 that contains the actual data records. The schema file maydefine the data structure for multiple patient conditions, including atleast one condition a current patient may be experiencing. The followinglisting is an example See5 based schema for a hypothetical test systemto diagnose one of the many potential conditions, a hypothyroidcondition:

diagnosis. age: continuous. sex: M, F. on thyroxine: f, t. query onthyroxine: f, t. on antithyroid medication: f, t. sick: f, t. pregnant:f, t. thyroid surgery: f, t. I131 treatment: f, t. query hypothyroid: f,t. query hyperthyroid: f, t. lithium: f, t. tumor: f, t. goitre: f, t.hypopituitary: f, t. psych: f, t. TSH: continuous. T3: continuous. TT4:continuous. T4U: continuous. FTI: = TT4/T4U. referral source: WEST,STMW, SVHC, SVI, SVHD, other. diagnosis: primary, compensated,secondary, negative. ID: label.

In the example schema, field variable names are in the left column, withtypes of values for variables specified in the right column. The field“age” has a continuous value indicative of the age of the patient. Thefield “sex” has a Male or Female designation. Other designations may beadded in further examples. Many other fields have true/false value typesshow as “f,t”. Other fields have values resulting directly from thefield name, such as the field “referral source”, which can have valuescorresponding to different doctors or health care facilities.

The example schema says that “diagnosis” is the dependent variable(which can take on the discrete values of primary, compensated,secondary, or negative) and all the other variables are the independentvariables. Here is an example of the input training data records basedon patient discharge data that correspond to the above schema for See5:

-   41,F,f,f,f,f,f,f,f,f,f,f,f,f,f,f,1.3,2.5,125,1.14,SVHC,negative,3733-   23,F,f,f,f,f,f,f,f,f,f,f,f,f,f,f,4.1,2,102,?,other,negative,1442-   46,M,f,f,f,f,N/A,f,f,f,f,f,f,f,f,f,0.98,?,109,0.91,other,negative,2965-   70,F,t,f,f,f,f,f,f,f,f,f,f,f,f,f,0.16,1.9,175,?,other,negative,806-   70,F,f,f,f,f,f,f,f,f,f,f,f,f,f,f,0.72,1.2,61,0.87,SVI,negative,2807-   64,F,f,f,f,f,f,f,f,f,f,f,f,f,f,f,?,?,?,?,other,negative,1821-   44,M,f,f,f,f,N/A,f,f,f,f,f,f,f,f,f,45,1.4,39,1.16,SVI,primary,2427-   48,M,f,t,f,f,N/A,f,f,f,t,f,f,f,f,f,5.4,1.9,87,1,other,negative,1433-   77,M,f,f,f,f,N/A,f,f,f,f,f,f,f,f,f,2,2.9,151,1.35,other,negative,942-   27,F,f,f,f,f,f,f,f,f,f,f,f,f,f,f,2.6,?,112,1.15,other,negative,3266-   65,M,f,f,f,f,N/A,f,f,f,f,f,f,f,f,f,14.8,1.5,61,0.85,SVI,compensated,251-   27,F,f,f,f,f,f,f,f,f,f,f,f,f,f,f,15,1.6,82,0.82,SVI,compensated,2691-   54,F,f,f,f,f,f,f,f,f,f,f,f,f,f,f,19,2.2,83,1.03,SVHC,compensated,822-   68,M,f,f,f,f,N/A,f,f,f,f,f,f,f,f,f,2.4,1.6,83,0.89,SVI,negative,2534-   . . .

The schema and the input are all in text format. The trained expertsystem produces a model, such as a decision tree in one embodiment.Other models may include deep neural network model, support vectormodels, and others. Some example expert systems may include See5, theDecisionTreeClassifier in a python package called scikit-learn, or the“tree” package in the R language to accomplish the same task. In furtherembodiments, other algorithms may be utilized to make predictions, suchas random forest and boosting algorithms. It is easy to construct adecision tree that includes a set of rules that will exactly match anexplicit set of data, but if new untrained data is later given, it willvery likely not match the rules well in the future. See5 tries toformulate rules that will model well (but probably not perfectly) theinput data, and that will also model well later for new data. This iswhat is meant by not overfitting the data. The data is matched closelywith the essence of the data, so that it will match future data aswell—but not perfectly. Note that in the rules, variables from thesource or input data are referred to as attributes.

Here are the results of running the full hypothyroid dataset through thetrained expert system 110:

Class specified by attribute {grave over ( )}diagnosis′ Read 2772 cases(24 attributes) from hypothyroid.data Decision tree: TSH <= 6: negative(2472/2) TSH > 6: :...FTI <= 65.3: primary (72.4/13.9)  FTI > 65.3: :...on thyroxine = t: negative (37.7)   on thyroxine = f:   :...thyroidsurgery = t: negative (6.8)    thyroid surgery = f:    :...TT4 > 153:negative (6/0.1)     TT4 <= 153:     :...TT4 <= 37: primary (2.5/0.2)     TT4 > 37: compensated (174.6/24.8) Evaluation on training data(2772 cases):    Decision Tree    ----------------    Size  Errors    7 10( 0.4%) <<    (a) (b) (c) (d) <−classified as    ---- ---- --------    60 −3    (a): class primary      154    (b): class compensated       −2 (c): class secondary     −4 −1  2548 (d): class negative  Attribute usage:     90% TSH     18% on thyroxine     16% thyroidsurgery     14% TT4     13% T4U     13% FTI Evaluation on test data(1000 cases):    Decision Tree    ----------------    Size  Errors    7  3( 0.3%) <<    (a) (b) (c) (d) <−classified as    ---- ---- --------    32    (a): class primary     −1 39   (b): class compensated        (c): class secondary     −1 −1  926 (d): class negative Time:0.0 secs

The decision tree is shown above. For the training data, the generateddecision tree produced a 0.4% error rate with 10 cases beingmisclassified out of the 2772 training cases. An evaluation test set of1000 cases was also run against the generated decision tree whichproduced a 0.3% error rate (3 misclassified out of the 1000 cases). Theevaluation data was not used to train the decision tree, simply to testit after creation. The decision tree may be implemented in linear codeto run data through it in an automated manner.

In one embodiment, training may be boosted. In one example, the modelcreated by training the expert system 100 is “boosted” with a10-instance collaboration. Boosting works by the system generatingdifferent instances of the decision tree following slightly differentgeneration philosophies. The final decision tree is then the combinationvoting of each different tree, where each have been individually run.The approach sometimes produces significant results. As in this case,the error rate on the training data is only 0.1%.

Some explanation would be useful for the format of errors that areproduced for either the training data or test data for the expert system110. The following shows the errors which are produced by the decisiontree for the above hypothyroid example:

 Decision Tree ---------------- Size  Errors  7 10( 0.4%) << (a) (b) (c) (d) <−classified as ---- ---- ---- ----  60 −3    (a):class primary   154    (b): class compensated      −2 (c): classsecondary  −4 −1  2548 (d): class negative

The decision tree size is the number of branches in the tree thatproduce a dependent variable answer (in this case 7). The number oferrors are the number of training/test cases that when run through therules produce an incorrect answer. For the above case there are 10,which is the sum of the four red numbers in the small table below it.That table, “classified as”, shows the 4 values of the dependentvariable and, on each row, how many training/test cases produce thecorrect output (positive numbers or in black) and how many produce theincorrect (negative numbers or in red) output.

Consider the values for “primary”. Sixty lines of training input whenrun through the rules produced the correct result of “primary”. Therewere, however, 3 lines of input that produced “compensated” (because itis on the “primary” line, but the “compensated” column) as the output,but it should have been “primary”. These 3 errors are thus type II(false-negative) errors relative to “primary”. The decision tree alsoshows that the rules produced incorrect answers of “primary” for 4training lines when those should have been “negative”. Thus, those 4cases where type I (false-positive) errors for “primary” but type II(false-negative) errors relative to “negative”.

In one example set of tests, length of stay (LOS) stories were used toevaluate the system utilizing a machine learning system, such as See5.The dependent variable (the output answer) to model rules for is this:did an individual patient discharge contribute to the generation of asignificant LOS story that shows that a hospital is out of the financialnorm for some pattern of patient clinical situations. The independentvariables are all the other patient attributes for a patient discharge.The variables starting with “PPC” followed by a number are codescorresponding to different conditions.

The following schema defines the patient attributes:

inStory. recordID: label. inStory: in, out. facility: ignore. APRDRG:discrete 317. medSurg: med, surg. serviceLine: discrete 41. soiLevel:low, high. SOI: U, L, M, H, X. eligible: f, t. actLOS: continuous.expLOS: continuous. cost: continuous. admittedICU: f, t. ICUdays:continuous. AnyPPC: f, t. PPCGroup1: f, t. | Extreme ComplicationsPPCGroup2: f, t. | Cardiovascular-Respiratory Complications PPCGroup3:f, t. | Gastrointestinal Complications PPCGroup4: f, t. | PerioperativeComplications PPCGroup5: f, t. | Infectious Complications PPCGroup6: f,t. | Malfunctions, Reactions, etc. PPCGroup7: f, t. | ObstetricalComplications PPCGroup8: f, t. | Other Medical and SurgicalComplications PPC01: f, t. | Stroke & Intracranial Hemorrhage PPC02: f,t. | Extreme CNS Complications PPC03: f, t. | Acute Pulmonary Edema andRespiratory Failure without Ventilation PPC04: f, t. | Acute PulmonaryEdema and Respiratory Failure with Ventilation PPC05: f, t. | Pneumonia& Other Lung Infections PPC06: f, t. | Aspiration Pneumonia PPC07: f, t.| Pulmonary Embolism PPC08: f, t. | Other Pulmonary Complications PPC09:f, t. | Shock PPC10: f, t. | Congestive Heart Failure PPC11: f, t. |Acute Myocardial Infarction PPC13: f, t. | Other Acute CardiacComplications PPC14: f, t. | Ventricular Fibrillation/Cardiac ArrestPPC15: f, t. | Peripheral Vascular Complications Except VenousThrombosis PPC16: f, t. | Venous Thrombosis PPC17: f, t. | MajorGastrointestinal Complications without Transfusion or SignificantBleeding PPC18: f, t. | Major Gastrointestinal Complications withTransfusion or Significant Bleeding PPC19: f, t. | Major LiverComplications PPC20: f, t. | Other Gastrointestinal Complicationswithout Transfusion or Significant Bleeding PPC21: f, t. | ClostridiumDifficile Colitis PPC23: f, t. | Genitourinary Complications ExceptUrinary Tract Infection PPC24: f, t. | Renal Failure without DialysisPPC25: f, t. | Renal Failure with Dialysis PPC26: f, t. | DiabeticKetoacidosis & Coma PPC27: f, t. | Post-Hemorrhagic & Other Acute Anemiawith Transfusion PPC28: f, t. | In-Hospital Trauma and Fractures PPC29:f, t. | Poisonings Except from Anesthesia PPC30: f, t. | Poisonings dueto Anesthesia PPC31: f, t. | Pressure Ulcer PPC32: f, t. | TransfusionIncompatibility Reaction PPC33: f, t. | Cellulitis PPC34: f, t. |Moderate Infectious PPC35: f, t. | Septicemia & Severe Infections PPC36:f, t. | Acute Mental Health Changes PPC37: f, t. | Post-ProceduralInfection & Deep Wound Disruption Without Procedure PPC38: f, t. |Post-Procedural Wound Infection & Deep Wound Disruption with ProcedurePPC39: f, t. | Reopening Surgical Site PPC40: f, t. | Peri-OperativeHemorrhage & Hematoma without Hemorrhage Control Procedure or I&DProcedure PPC41: f, t. | Peri-Operative Hemorrhage & Hematoma withHemorrhage Control Procedure or I&D Procedure PPC42: f, t. | AccidentalPuncture/Laceration During Invasive Procedure PPC44: f, t. | OtherSurgical Complication - Moderate PPC45: f, t. | Post-Procedural ForeignBodies PPC46: f, t. | Post-Procedural Substance Reaction & Non-O.R.Procedure for Foreign Body PPC47: f, t. | Encephalopathy PPC48: f, t. |Other Complications of Medical Care PPC49: f, t. | IatrogenicPneumothorax PPC50: f, t. | Mechanical Complication of Device, Implant &Graft PPC51: f, t. | Gastrointestinal Ostomy Complications PPC52: f, t.| Infection, Inflammation & Other Complications of Devices, Implants orGrafts Except Vascular Infection PPC53: f, t. | Infection, Inflammationand Clotting Complications of Peripheral Vascular Catheters andInfusions PPC54: f, t. | Central Venous Catheter-Related Blood StreamInfection PPC59: f, t. | Medical & Anesthesia Obstetric ComplicationsPPC60: f, t. | Major Puerperal Infection and Other Major ObstetricComplications PPC61: f, t. | Other Complications of Obstetrical Surgical& Perineal Wounds PPC63: f, t. | Post-Procedural Respiratory Failurewith Tracheostomy PPC65: f, t. | Urinary Tract Infection PPC66: f, t. |Catheter-Related Urinary Tract Infection ageGroup: under 65, 65 to 74,75 to 84, over 84. weekendAdmit: f, t. mondayDisch: f, t. PACfacility:Home, HH, LTAC, Rehab, SNF.

“inStory” is the dependent variable that can have the values “in” or“out”, meaning that this discharge (each line of data represents asingle discharge) either contributed to a significant story (“in”) or itdidn't (“out”). An example of an in story is a length of stay thatexceeded an average length of stay for a particular type of treatmentthat is normal for a particular healthcare facility. In other words,there is likely a story regarding why the patient had a longer stay thanaverage. The average in various embodiments may be calculated for eithera group of healthcare facilities run by an organization, or even anexemplary healthcare facility, such as a facility that has the bestlength of stay average of the group.

All the other variables are independent (even if they are truly not,they are considered to be) variables with various types and values. Someare of type “continuous” which means that they can have any numericvalue. Some have discrete values like “f” or “t” (false or true) or somenumber of discrete values that are not explicitly designated. An exampleof this is “APRDRG: discrete 317.” In this case, 317 unique DRGs are theinputs with values not explicitly defined.

Here is an example of the data that is being used to train the expertsystem that conforms to the schema defined above:

-   3,in,1,691,med,267,high,X,t,33,15.3,33630.21,t,32,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,75to84,f,f,Home-   4,out,1,463,med,250,low,M,t,2,3.3,--   2074.21,t,2,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,f,75to84,f,f,Home-   5,out,1,5,surg,132,high,X,f,48,31.2,47623.08,f,0,t,f,f,t,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,t,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,65to74,f,t,Home-   6,in,1,139,med,296,high,H,t,16,5.3,18313.354,2,t,f,f,f,f,t,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,-   7,out,1,440,surg,387,high,H,t,7,8.1,--   7780.65,t,4,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,f,under65,f,f,Home-   8,out,1,315,surg,274,low,M,t,2,2.5,--   3017.95,f,0,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,f75to84,f,f,HH-   9,out,1,260,surg,132,high,H,t,8,10.5,--   6780.92,t,3,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,f,75to84,f,f,HH-   10,in,1,321,surg,274,low,L,t,8,1.6,42241.93,f,0,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,under65,f,f,Home-   11,out,1,120,surg,133,high,H,t,8,11.0,--   8036.00,t,8,t,f,t,f,f,f,f,f,f,f,f,t,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,75to84,f,f,HH-   12,out,1,301,surg,274,high,H,t,3,5.0,--   9297.61,t,3,t,f,f,f,f,f,f,f,t,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,t,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,75to84,f,f,SNF-   13,out,1,55,med,255,low,M,t,6,3.3,5987.63,t,6,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,    f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,75to84,f,f,SNF

For this data the “recordID” is just a sequential number and does notcorrespond to any real identifier. “facility” is always the number 1, sothere is no PHI in these records.

FIG. 2 is a flowchart illustrating a computer implemented method 200 ofpredicting resource utilization in the treatment of patients accordingto an example embodiment. At operation 210, a schema is obtaineddefining variables, also referred to as attributes in the resultingmodel, representing patient treatment experiences for one or more healthconditions. Data is obtained at operation 220. The data includes datafrom historical patient treatment experiences and is arranged inaccordance with the schema. A machine learning system is trained atoperation 230 based on a training set of the data for a specifiedtime-period. At operation 240, a model, such as a decision tree isgenerated from the trained machine learning system. The decision tree isgenerated based on attributes selected by the trained machine learningsystem to predict patient treatment experiences for the one or morehealth conditions.

The decision tree may be pruned as a function of attribute usage in thedecision tree. The decision tree may also be boosted by generatingmultiple different decision trees from the trained machine learningsystem and combining results from the multiple different decision treesto predict patient treatment experiences for the one or more healthconditions.

In some embodiments, the data may be stored in a database in accordancewith a schema defining the data including a dependent variable name andcorresponding discrete value selected from a set of discrete values, afirst set of multiple independent variables having correspondingdiscrete values, and at least one independent variable having acorresponding continuous value.

The treatment experiences in one example includes a length of staycorresponding to a patient treatment procedure. The data may correspondto a single healthcare facility, and the specified time-period mayinclude a time-period starting from a time following a change inprocedures at the single healthcare facility.

FIG. 3 is a flowchart illustrating a method 300 of evaluating theeffectiveness of changes made to procedures in the healthcare facility.The changes may be generated based on analysis of the training data byother systems, or by human analysis in various embodiments. In method300, the decision tree may be used at operation 310 to generate a firstlength of stay statistic for a first time-period prior to the change inprocedures and also to generate at operation 320 a second length of staystatistic for a second time-period following the change in procedures.Note that in some embodiments, expert system 110 is trained separatelyfor each of the different time-periods, and a corresponding model, suchas a decision tree, is generated for each different time-period.Operation 330 is used to determine the effectiveness of the change inprocedures as a function of the length of stay statistics. If the lengthof stay is about the same, the changes likely did not help improveresource utilization. If the length of stay increased, the changes werelikely counterproductive. If the length of stay decreased over thesecond time-period, the changes were likely very effective. In anyevent, further changes may be tried with successive measurements takenfor each further change to measure effectiveness in resourceutilization. The length of stay statistic may comprise a percent ofpatient treatment experiences that are above a selected threshold overan average length of stay for a respective time-period.

FIG. 4 is a flowchart illustrating a method 400 of generating a furtherimproved decision tree. At operation 410, the trained machine learningsystem may be tested on a set of test data from the obtained data fromhistorical patient treatment experiences. Test data patient treatmentexperiences may be classified via the decision tree at operation 420. Atoperation 430, the number of patient treatment experiences misclassifiedby the decision tree are identified as “in story”. Operation 440identifies the number of patient treatment experiences correctlyclassified by the decision tree as “out of story”. Operation 450generates a new decision tree by using cost qualifiers based onincorrectly classified in story patient treatment decisions to increaseaccuracy of the decision tree.

In some embodiments, machine learning systems provide the ability toexecute produced decision trees against new rows of data. In oneexample, See5 packages come with a rule execution module that can beintegrated into other software which can run the generated rules againstnew data rows.

If the data for a particular patient is not complete, the rules maystill be executed in some embodiments. Perhaps a full coded record isnot available until a formal coding is performed after the patient hasbeen discharged. Even if this is the case, a preliminary diagnosis willbe provided for the patient (which could change over time). Thatdiagnosis and any subsequent diagnoses may be grouped to produce apreliminary DRG (diagnosis related group) while the patient is in thehospital. That data (along with any subsequent data) can be fed into thetrained expert system to determine if the patient has a potential riskof going out of the norm for cost LOS (length of stay). Rules are thengenerated with the level of data that is available when the rules arerun.

For instance, if doctors' notes at a facility are all auto-coded as soonas they are entered (prior to a final coding session by a coder), then atentative DRG could be determined. MedSerg, serviceLine, SOI, LOS (up tothat current point), admittedICU, ICUdays (to the current point),ageGroup, and weekendAdmit could also be determined from the date ofadmittance to the evaluation time. This data could be executed (andperhaps re-executed again and again) against the rules for a site to seeif that data indicates a potential out-of-norm cost in the future.

A data store, such as database 125, associated with the rules couldbegin to monitor utilization even if the data is run through the rulespost-discharge but before the system 100 has a chance to process thedata. This could be done with a coding system, such as 3M™ 360Encompass™ system, so as soon as codes are assigned and DRG groups areassigned, the rules could be run to determine out-of-norm costs, and adata store per facility could collect the types and categories ofoverruns.

In some examples, patient discharge records may be analyzed againststories where the facility or enterprise is doing well. While storiesare classified as in or out of norm in the above examples andembodiments, stories may also be gathered for a whole spectrum of “doingwell to doing poorly.” Rules may be generated to show not only where thefacility is doing poorly but also where it is excelling. Armed with thatinformation, a facility may be better able to plan utilization ofresources.

In one example, the system 100 runs a hospital's data through itsmeasures and other code or humans may make story recommendations basedon the results. The results capture the performance for a first periodof time prior to implementation of changes based on the recommendations.After waiting a second period of time sufficient for the changes to takeeffect and collect a sufficient number of patient treatments fortraining and testing, data corresponding to the second time-period maybe run to again and produce another set of stories—stories telling thehospital where their resource utilization out-of-norm problems are.Maybe the problems will be the same ones because they haven't fixed theones they were previously told about, or maybe they will be a whole newslew of story problems.

In one embodiment, system 100 rules may be continuously running, evenlong after a patient has been discharged, to give that facility acumulative view of the effect of working on the system-exposed problems.These rules could be running, as a matter of course, against dischargedpatient records to accumulate information and provide an up-to-datedashboard of how the facility is doing relative to the latest set ofrules. Since discrete data is being collected, drilling down into thosediscrete data provides evidence of facility performance change.

When a new set of data is run through the system, a new set of rules isgenerated that reflects an up-to-date view of where the facility'sproblems are and how they are executing against them over time.

In one example analysis, a decision tree is generated for a firstexample set of provider data. The decision tree has a cost ratiodesignated that says that an “out” (in of norm)-designated story thatshould be “in” the story is 5 times worse than an “in”-designated storythat should be “out”-designated. This, of course all depends on how therules are being used. The error rate for this decision tree is 1%,meaning that 1% of the cases are misclassified.

Evaluation Statistics:

  Evaluation on training data (9817 cases):   Decision Tree----------------------- Size  Errors Cost  423 103( 1.0%) 0.01 << (a) (b) <−classified as ---- ---- 2492   (a): class in  103 7222 (b):class out  Options:  Pruning confidence level 99%  Class specified byattribute {grave over ( )}inStory′  Read 9817 cases (84 attributes) fromLOS_Test1_Story_060818.data  Read misclassification costs fromLOS_Test1_Story_060818.costs

The following is an example first small portion of the correspondingDecision tree:

eligible = f: out (518) eligible = t: :...admittedICU = f: :...PPCGroup5 = t: in (42)  : PPCGroup5 = f:  : :...PPC24 = t:  : :...medSurg = med: in (21)  :  : medSurg = surg: out (18)  :  PPC24 =f:  :  :...PPC48 = t: in (4/1)  : PPC48 = f:  : :...PPCGroup6 = t:  : :...medSurg = med: in (10)  :  : medSurg = surg: out (6)  :  PPCGroup6= f:  :  :...PPCGroupl =t: in (1)  : PPCGroup 1 = f:  : :...APRDRG in691,5,120,56,163,20,6,196,190,4,130,162,  :  :57,401,165,176,910,753,2,166,170,1,40,  :  :90,774,110,770,511,911,382,841,650,51,  :  : 138: out (0) ...

The actual tree is much longer in length. The portion provided isrepresentative of the longer tree produced by the machine learningsystem based on the data provided. Since data from different entitiesand different entity time-periods will vary, resulting decision treeswill also vary.

   Evaluation on training data (9817 cases):   Decision Tree----------------------- Size Errors Cost  423 103( 1.0%) 0.01 << (a) (b) <−classified as ---- ---- 2492   (a): class in  103 7222 (b):class out        Attribute usage:  100% eligible   95% admittedICU   94%PPCGroup6   91% APRDRG   55% PPC47   54% PPC27   39% PPCGroup5   38%PPC24   38% PPCGroup1   30% PPC48   24% soiLevel   19% AnyPPC   10%expLOS    4% PPCGroup3    4% ageGroup    4% SOI    3% PACfacility    3%PPCGroup2    2% weekendAdmit    2% PPCGroup8

In a second example, boosting of the data of the first example isillustrated. Boosting works by the system generating different instancesof the decision tree following slightly different generationphilosophies. The final decision tree is then the combination voting ofeach different tree, where each have been individually run. The approachsometimes produces significant results. As in this case, the error rateon the training data is only 0.1%.

The following shows the results of “boosting” the data. Boosting causesthe system to generate different decision trees for the same data. Thedifferent error rates for each of the generated decision trees may beobserved, some being very good and others quite poor. The final“boosted” decision tree is subsequently not a tree per-se but the voting(and running by) of each tree against a data set. This process, as youcan see below, often produced excellent results.

   Options: 10 boosting trials Pruning confidence level 99%  Classspecified by attribute {grave over ( )}inStory’  Read 9817 cases (84attributes) from  LOS_Testl_Story_060818.data  Evaluation on trainingdata (9817 cases):  Trial Decision Tree  ----- ----------------  Size Errors   0 328 125( 1.3%)   1 510 499( 5.1%)   2 525 466( 4.7%)  3 750 913( 9.3%)   4 697 375( 3.8%)   5 648 353( 3.6%)   6 5361652(16.8%)   7 587 893( 9.1%)   8 699 756( 7.7%)   9 547 558( 5.7%) boost   7( 0.1%) <<  (a) (b) <−classified as  ---- ----  2490 2 (a):class in  5 7320 (b): class out

A third example contains yet another modification to the process for thefirst example data. In this example, the “cost” of an “out” (out of thestory) record that should have been designated as an “in” (in the story)record (type II error relative to “in”) to be 100 times costlier than an“in” record that should have been designated as an “out” record. Thissays that if a record is in the story (story meaning that these recordsare out of the financial norm), it's bad to say that it is out of thestory. The opposite may not be as bad depending on what is beingmeasured. Whether certain kinds of error are found as “bad” or notdepends entirely on how this technology may be used in the future. Fordifferent customers, a 90 to 95% correct prediction that a patient hasor may go out of the norm may or may not be acceptable. The actual errorrate obtained by the system may depend on the amount of data and qualityof the data used to train the system. Such error rates may be less thanor greater than the rates referenced herein.

Options:  Pruning confidence level 99%  Test requires 2 branches with >=1 cases   Class specified by attribute {grave over ( )}inStory’   Read148442 cases (84 attributes) from file   Read misclassification costsfrom costs file   Evaluation on training data (148442 cases):   Decision Tree   -----------------------   Size  Errors Cost   27574179( 2.8%) 0.03 <<   (a) (b) <−classified as   ----- -----  2381   (a): class in   4179 141882 (b): class out

A fourth example contains a decision tree that was generated for thefirst set of a second provider data. The error rate for that data is6.7%. This decision tree was tested with data from the second set ofsecond provider data and it produced 4.3% errors, the majority of thesebeing “in” records that were misclassified as “out” which is thought tobe a more-severe form of error. The test was also reversed with thedecision tree built with the second set of data and tested it with thefirst.

A fifth example shows those results with essentially a 0% error rate forthe second provider 2 training data. The second provider 1 data runagainst this tree did produce a 12.1% error rate, but the majority ofthese were type I errors.

In this evaluation the below decision tree was built from the first setof second provider data that was received. The error rate was 6.7%, butthese were all type I errors. The error rate for the test set of thesecond set of second provider is less at 4.3%, but most of these aretype I errors. There were 21,601 training cases. Then, as part of therun, 5,186 discharge cases were processed from the second set of secondprovider data.

Evaluation Statistics:

     Evaluation on training data (21601 cases):    Decision Tree  -----------------------   Size  Errors Cost    116 1451( 6.7%) 0.07 <<   (a) (b)  <−classified as   ---- ----    622  (a): class in   145119528 (b): class out      Evaluation on test data (5186 cases):   Decision Tree   -----------------------   Size  Errors Cost   116 221( 4.3%)  1.95 <<    (a) (b)  <−classified as   ---- ----   570 202  (a): class in   19 4395  (b): class out Options:  Pruningconfidence level 99%   Class specified by attribute {grave over( )}inStory’   Read 21601 cases (84 attributes) from LOS_second providerdata   Read misclassification costs from LOS second provider data costs     Decision tree excerpt:   PPCGroup1 = t: out (2081)   PPCGroup1 = f:  :...eligible = f: out (1690)    eligible = t:    :...expLOS <= 2.8:    :...cost <= 1701.89:     :...expLOS <= 1.5:     : : :...serviceLinein 165,296,90,262,294,250,293,95,45,125,    : : : :   274,57,283,276,145,70,135,267,387,330,    : : : :   390,137,272,269,133,65,252,85,170,58,     : : : :   1: in(0)     : : : serviceLine in 245,50,132,255,280: out (73)    : : : serviceLine = 129:     : : : :...weekendAdmit = f: in (12/8)    : : :  weekendAdmit = t: out (4)     : : expLOS > 1.5:      ...

In a fifth example, the opposite of the fourth example was done. Adecision tree is built with the second provider 2 data, and is testedwith the second provider 1 data set. The results are a very simpledecision tree that produces only 2 type I errors and no type II errorsover the 5,186 training cases. Testing the first set of Second providerdata (21,601 cases) does produce a 12.1% error rate, but most of theseare type I errors.

    Options:   Pruning confidence level 99% Class specified by attribute{grave over ( )}inStory’ Read 5186 cases (84 attributes) from LOS_SecondproviderMemorialHospital330235_Story_061918.data Read misclassificationcosts from LOS_Second providerMemorialHospital330235_Story_061918.costsDecision tree: eligible = f: out (468) eligible = t: :...APRDRG in770,955,44,662,131: out (0)  APRDRG in640,773,425,560,135,253,244,313,403,197,461,460,383,756,141,540, :   222,52,720,140,194,54,139,247,45,221,241,47,566,284,663,351,710, :   952,58,380,861,757,309,248,263,198,776,446,463,422,225,240,204, :   342,137,249,134,227,136,465,199,251,224,341,346,424,468,541,382, :   282,626,816,308,951,850,245,723,143,317,203,220,466,207,283,51, :   892,41,53,254,347,115,581,246,691,279,113,82,890,364,243,281, :   111,228,531,501,196,660,42,421,513,694,144,344,340,724,252,130, :   223,614,811,264,49,262,320,190,519,950,142,760,752,813,43,462, :   774,361,314,205,349,138,229,542,23,500,315,482,532,384,200,362, :   171,740,363,423,280,261,193,791,546,711,721,661,48,242,862,722, :   633,305,381,514,40,55,518,755,385,50,758,26: out (3895)  APRDRG in750,420,201,302,754,301: in (446/2)  APRDRG= 133:  :...soiLevel = low:out (11)    Evaluation on training data (5186 cases):   Decision Tree  -----------------------   Size  Errors Cost    13 2( 0.0%) 0.00 <<  (a) (b) <−classified as   ---- ----   772   (a): class in   2 4412 (b): class out      Attribute usage:   100% eligible    91%APRDRG    7% soiLevel     Evaluation on test data (21601 cases):  Decision Tree   -----------------------   Size  Errors Cost    132616(12.1%) 0.12 <<   (a) (b) <−classified as   ---- ----   601 21 (a):class in   2595 18384 (b): class out

A sixth example shows a decision tree for a third example facility data.The decision tree is odd because everything is declared as “out.” Thisis because there were just 330 discharges out of 93,389 which weredeemed to be “out of the norm” (‘in’ the story) for LOS. It appears thatthe third facility is doing a good job managing LOS. The system can bemanipulated to create a decision three that will eliminate the ‘in’sthat show up in the ‘out’s state by adding the costs factor. The errorrate for this is 0.9%, however, the generated decision tree is much morecomplex. Out of the 93,389 discharge cases only 330 were “in” a story,or in other words out of the norm financially. Thus, there was less than0.4% error rate for LOS management in these cases. This shows goodlength-of-stay management for the third facility.

    Options:  Focus on errors (ignore costs file) Class specified byattribute {grave over ( )}inStory’ Read 93389 cases (84 attributes) fromLOS_Third Facility_Story_061818.data Decision tree:  out (93389/330)Evaluation on training data (93389 cases):   Decision Tree  ----------------   Size  Errors    1 330( 0.4%) <<  (a) (b)  <−classified as   ---- ----     330  (a): class in   93059 (b): class out

In a seventh example, an evaluation of the third facility data uses a“costs” file to designate that “ins” that were produced as “outs” were100 times worse than not designating them that way (thus the cost).Using the “costs” qualifiers tens to create a more complex decisiontree. The cost of eliminating these type II errors is a 0.9% error rate,but these come from “outs” that were misclassified as “ins”.

Evaluation Statistics:

Evaluation on training data (93389 cases):   Decision Tree  -----------------------   Size  Errors Cost    366 862( 0.9%) 0.01 <<   (a) (b) <−classified as   ---- ----    330   (a): class in    86292197 (b): class out     Options:  Pruning confidence level 99% Classspecified by attribute {grave over ( )}inStory’ Read 93389 cases (84attributes) from third facility data Read misclassification costs fromthird facility costs data Decision tree excerpt: PPC40 = t: :...medSurg= med: out (32) : medSurg = surg: : :...eligible = f: out (24):  eligible = t: :  :...PPC08 =t: out (8) :   PPC08 = f: :   :...PPC01 =t: out (6) :    PPC01=f: :    :...PPC50 = t: out (6) :     PPC50 = f::     :...APRDRG in 44,720,201,248,247,139,711,425,384,137,190,:      :   111,812,660,691,194,460,383,135,45,347,249,:      :   134,52,133,140,254,93,22,143,207,663,466,:      :   206,110,141,197,58,241,463,420,279,191,136,:      :   113,252,721,501,243,42,204,753,43,791,566,:      :   540,144,53,253,750,403,861,130,461,55,693,:      :   199,246,284,49,240,196,245,56,244,468,200,:      :   314,41,813,282,47,203,142,661,176,260,351,:      :   192,283,756,465,380,346,343,951,754,198,424,:      :   21,5,115,842,385,757,930,751,23,138,722,514,:      :   444,264,446,775,48,445,344,242,681,422,222,       Evaluationon training data (93389 cases):    Decision Tree  -----------------------   Size  Errors Cost   366 862( 0.9%) 0.01 <<  (a) (b) <−classified as  ---- ----   330   (a): class in   86292197 (b): class out  Attribute usage:   100% PPC40   100% medSurg  100% PPC65    72% PPC10    71% PPC16    71% PPC53    71% PPC20    71%PPC52    71% PPC23    71% PPC48    71% PPC19    71% PPC29    71% PPC17   71% service Line    38% AnyPPC    26% APRDRG    10% expLOS    4% cost   3% actLOS    2% PACfacility    2% ageGroup    2% admittedICU    2%eligible    1% weekendAdmit    1% soiLevel

An eight example, contains a decision tree for data from one of a fourthentities clinics. This generation came from 770,768 discharge cases.Since the goals of See5 are to reduce the error rate (0.5% for thiscase) as well as reduce the complexity of the decision tree, this dataachieves that goal. There are many more Type II errors (again ‘in’s inthe ‘out’s category) in this tree than Type I errors (‘out’s in the‘in’s category). If the cost of Type II errors is greater, costparameters can be added to create a tree with these reduced.

Data from one of the fourth entity's facilities. The default decisiontree that is generated is very simple. It produced a (less than) 0.5%error rate, again mostly composed of discharges that were “in” whichwere designated as “out” (type II errors). Again, given the low numberof discharges contributing to “in” stories (3823) versus the totalnumber of discharges (766945), this fourth entity facility seems to bedoing well with its LOS management.

     Options:   Focus on errors (ignore costs file) Class specified byattribute {grave over ( )}inStory’ Read 770768 cases (84 attributes)from LOS_FourthEntity_Story_061818.data Decision tree excerpt: PPC37 =t: out (1072/63) PPC37 = f: :...ICUdays <= 20: out (765920/3624) ICUdays > 20:  :...facility in 3,4,5,6,7,8: out (2832/5)   facility =1:   :...expLOS <= 33.8: out (392/4)   : expLOS > 33.8:   : :...expLOS<= 36.1: in (14/3)   :   expLOS > 36.1: out (66)   facility = 2:  :...expLOS <= 36.1: out (406/69)    expLOS > 36.1:    :...weekendAdmit= t: in (16)     weekendAdmit = f:     :...actLOS <= 45: out (28/10)    actLOS > 45: in (22/1)      Evaluation on training data (770768cases):   Decision Tree   ----------------   Size  Errors   10 3787(0.5%) <<   (a) (b) <−classified as   ----- -----   40 3783 (a): class in   4 766941 (b): class out  Attribute usage:   100% PPC37   100% ICUdays  35% facility    9% expLOS

A ninth example shows just the resulting statistics of adding the costparameters to reduce the Type II errors. The generated decision tree islong and complex, so it is not shown, but as you can see the error rateis still a low 1%. With this clinic analysis, the type II errors for“in” have been reduced, but not eliminated, and there is a 1% overallerror rate. Again, the type II error rate has been reduced by specifyingthat “ins” that have been specified as “outs” is 100 times the cost ofnot producing that error. The actual decision tree is quite complex andadds nothing by showing it here.

   Evaluation on training data (770768 cases):   Decision Tree----------------------- Size  Errors Cost 3291 7869( 1.0%) 0.01 << (a) (b) <−classified as ----- -----  3816  7 (a): class in  7862759083 (b): class out

In a tenth example, the first example data is revisited. The amount ofdata that went into producing them was greatly reduced. Information waseliminated to include only information that would be available from apatient stay in the training data, built the new rules, and tested them.This produced a 5.4% error rate (mostly type I errors) as opposed to the1% that the original Tampa data produced. This does seem reasonablegiven the amount of data that was not available for rule generation.

This evaluation shows the results of running the system with the greatlyreduced dataset. The following are the only fields that this decisiontree has been trained on:

recordID: label. inStory:  in, out. APRDRG: discrete 317. medSurg: med,surg. serviceLine: discrete 41. soiLevel: low, high. SOI: U,L,M,H,X.eligible:  f,t. actLOS:  continuous. expLOS:  continuous. admittedICU:f,t. ICUdays: continuous. ageGroup: under65, 65to74, 75to84, over84.weekendAdmit: f,t.

The intent of this data is to reduce the fields to only those that maybe available during a patient stay. Again, this absolutely presumes thatconcurrent auto-coding and DRG generation are performed while thepatient is in the hospital or in a timely manner based on the use of thedata.

Here are the evaluation statistics:

Evaluation on training data (9817 cases):   Decision Tree  -----------------------   Size  Errors Cost   730 530( 5.4%) 0.07 <<  (a) (b) <−classified as   ---- ----   2464 28 (a): class in  502 6823 (b): class out  Attribute usage:    100% eligible    95%APRDRG    95% admittedICU    49% expLOS    31% soiLevel    18% actLOS   18% ageGroup    12% ICUdays    10% weekendAdmit     8% SOI

Though the decision tree is quite long, the use of the DRG early in theevaluation is of note. As can also be seen above the DRG is used in 95%of the decision tree logic.

  Options:    Pruning confidence level 99%    Test requires 2 brancheswith >= 1 cases Class specified by attribute {grave over ( )}inStory’Read 9817 cases (84 attributes) from LOS_Test1_Story_060818.data Readmisclassification costs from LOS_Test1_Story_060818.costs Decision treeexcerpt: eligible = f: out (518) eligible = t: :...admittedICU = f: :...expLOS <= 2.2:  : :...APRDRG in50,194,691,463,5,139,440,260,120,301,55,56,141,720,420, : : :   791,312,45,756,383,173,302,346,952,180,140,711,384,263, : : :   42,460,44,174,136,254,163,199,815,137,191,244,281,261, : : :   143,951,662,710,52,197,466,264,169,221,58,133,248,468, : : :   20,950,842,241,253,46,724,844,681,80,308,171,309,246, : : :   364,6,226,196,227,813,252,282,48,443,344,303,21,284,660, : : :   224,247,92,755,447,23,380,190,243,347,757,279,283,349, : : :   167,220,4,313,721,385,775,351,223,444,424,530,229,130, : : :   134,161,850,121,240,930,162,912,442,305,57,314,43,401, : : :   441,540,462,304,425,423,165,693,176,722,381,694,361,245, : : :   910,860,753,262,340,723,501,2,862,166,170,680,461,142, : : :   531,341,135,193,843,1,225,40,280,894,205,91,26,317,892, : : :   421,651,90,774,690,22,41,110,770,343,661,518,511,751, : : :   177,911,89,776,563,773,510,131,561,382,890,49,405,893, : : :   841,517,750,500,650,692,51,138: in (0)  : : APRDRG in315,316,201,82,73,115,251,198,175,812,204,113,24,422, : : :   203,310,320,206,54,114,111,519,144,47,207,192,861,249, : : :   465,404,98,222,566,560,70,512,532,242,513,811,362,200, : : :   363,97,93,228,484: out (131)  : : APRDRG in53,816,446,342,403,481,482,445,480: in (53)  : : APRDRG = 663: : : :...ageGroup = under65: in (1)  : : : ageGroup in65to74,75to84,over84: out (3)       Evaluation on training data (9817cases):    Decision Tree   -----------------------   Size  Errors Cost  730 530( 5.4%) 0.07 <<   (a) (b) <−classified as   ---- ----  2464 28 (a): class in   502 6823 (b): class out  Attribute usage:  100% eligible    95% APRDRG    95% admittedICU    49% expLOS    31%soiLevel    18% actLOS    18% ageGroup    12% ICUdays    10%weekendAdmit    8% SOI

In an eleventh example, the second provider data is revisited bybuilding the rules from the second provider 1 data set again, butbuilding them using the same reduced schema used above. That data andthe second provider 2 data was run through the rules comprising themodel/decision tree. Surprisingly the training data (second provider 1)produced an 8.4% error rate (mostly type I errors) while the secondprovider 2 test data (reduced as well) produced only a 4.1% error rate.

In this test, rules were generated utilizing the reduced set of Secondprovider 1 data. Statistics are shown for that generation and alsoSecond provider 2 data run against these reduced set of rules.

Test Statistics:

Evaluation on training data (21601 cases):    Decision Tree  -----------------------   Size  Errors Cost   61 1822( 8.4%) 0.08 <<  (a) (b) <−classified as   ---- ----   622  (a): class in   182219157 (b): class out  Attribute usage:   100% eligible    91% expLOS   63% APRDRG    29% serviceLine    9% soiLevel    7% ageGroup    5%admittedICU    2% actLOS    2% weekendAdmit     Evaluation on test data(5186 cases):     Decision Tree   -----------------------  Size  Errors Cost   61 213( 4.1%) 3.74 <<   (a) (b) <−classified as  ---- ----   578 194 (a): class in   19 4395 (b): class out

This is interesting that the secondary data produced a smallerpercentage of errors though they were mostly of type II errors. Here isthe decision tree and all statistics for the run:

   Options:    Pruning confidence level 99% Class specified by attribute{grave over ( )}inStory’ Read 21601 cases (84 attributes) fromLOS_Second provider data Read misclassification costs from LOS_Secondprovider costs data Decision tree excerpt: eligible = f: out (1880)eligible = t: :...expLOS <= 2.8:  :...serviceLine in294,293,267,269,133,252,85,170,58,1: out (0)  : serviceLine in245,165,296,90,50,262,250,95,45,125,132,274,255,57,283, : :    276,280,145,70,135,387,330,390,137,272,  : :    65: out(6214) : serviceLine = 129:  : :...APRDRG in640,383,139,425,720,140,201,133,560,774,468,750,420,773, :  :   190,204,540,247,135,253,244,313,42,952,171,710,463,282, :  :   113,194,137,142,308,143,753,130,302,341,460,754,224,115, :  :   249,566,751,198,46,347,45,111,951,207,254,403,54,141, :  :   663,55,380,513,144,252,44,281,284,136,775,757,446,776, :  :   351,722,589,245,263,301,251,342,755,248,53,421,279,581, :  :   241,723,221,206,197,47,484,660,196,423,317,243,465,283, :  :   225,950,690,246,531,52,385,721,315,422,82,48,134,203, :  :   340,57,691,349,384,58,138,304,240,424,500,223,227,843, :  :   791,226,694,242,364,228,343,51,346,661,626,309,80,756, :  :   305,49,344,199,614,561,758,110,541,50,466,752,310,760, :  :   518,381,176,262,724,314,321,633,200,483,532,461,514,501, :  :   41,442,563,681,43,441,930,280,512,316,565,445,361,220, :  :   564,180,912,510,222,519,404,170,121,205,320,229,544,382, :  :   711,131,114,443,636,405,362,26,650,98,892,546,639,56, :  :   193,770,482,191,530,862,850,890,264,462,955,542,23,740, :  :   363,261,662,40,680,167,894,447,517,260,  :  :   844: in (0) :  APRDRG in 861,816,815,813,811: out (42)  :  APRDRG = 812:    ...    Evaluation on training data (21601 cases):    Decision Tree  -----------------------   Size  Errors Cost    61 1822( 8.4%) 0.08 <<  (a) (b) <−classified as   ---- ----   622   (a): class in   182219157 (b): class out  Attribute usage:    100% eligible    91% expLOS   63% APRDRG    29% serviceLine     9% soiLevel     7% ageGroup     5%admittedICU     2% actLOS     2% weekendAdmit     Evaluation on testdata (5186 cases):    Decision Tree   -----------------------  Size  Errors Cost    61 213( 4.1%) 3.74 <<   (a) (b)  <−classified as  ---- ----   578 194 (a): class in    19 4395 (b): class out

In a twelfth example, the results of running the latest second providerdata (Second provider 3) through the system are shown. The error rate isa very small 0.1%. Of particular note is that of the 5,164 Patientdischarges that were included in this data set 34% (1309) of thepatients contributed to the “in” story. Or in other words, 34% of thesehad LOSs which were out of the norm. This hospital surely needs help.

In this evaluation, the latest second provider data is processed toproduce a simple decision tree that models that latest data with a 0.1%error rate. As with the other second provider data, all produced simpledecision trees. This set of data possessed 34% of the records that fedinto the story (out of norm) which is high. There is definite room forLOS improvement with this site.

     Options:   Pruning confidence level 99% Class specified byattribute {grave over ( )}inStory’ Read 5164 cases (84 attributes) fromLOS_SECOND PROVIDER data Decision tree excerpt: expLOS <= 2: :...expLOS<= 1.7: out (32/2) : expLOS > 1.7: : :...serviceLine in296,165,294,250,245,293,90,267,274,57,330,276,145,95,:  :   137,252,70,272,85,65,170,133,58: in (0) :  serviceLine in262,132: in (316/1) :  serviceLine in129,125,387,255,283,135,45,280,390: out (34) :  serviceLine = 50::  :...admittedICU = f: out (12) :   admittedICU = t: in (43) expLOS >2: :...admittedICU = f:  :...expLOS <= 7:  : :...PPCGroup8 =t: : : :...medSurg = med: out (4)  : : : medSurg = surg: in (12) : : PPCGroup8 = f:  : : :...APRDRG in750,710,500,757,203,130,49,252,362,317,57,951,650, : :  :   912,723,950,364,305,82,46,98,465,791,343,565,205, : :  :   892,441,681,811,815,952,50,309,581,280,224,115,544, : :  :   310,691,930,382,121,442,519,443,191,445,      : :  :   862:out (0)     Evaluation on training data (5164 cases):   Decision Tree  ----------------   Size  Errors   44 7( 0.1%) <<   (a) (b) <−classified as   ---- ----   1305  3 (a): class in   4 3852  (b): class out  Attribute usage:   100% expLOS    93%admittedICU    64% APRDRG    55% PPCGroup8    40% serviceLine    12%soiLevel     4% eligible

A thirteenth example shows the same latest second provider data (secondprovider 3) but processed by a 10-fold cross-validation of the data.What this means is that 10 random subsets of the data are taken where,in this case, 75% of the data is used for training a decision tree and25% of the subset is used to test the trained decision tree. As wassaid, 10 of these are created and tested and the resulting statisticsare averaged together to give an estimated average error rate. The meanerror rate turned out to be 2.3% with a standard error of 0.3%. Thesenumbers are quite a bit higher than the “untested” 0.1% error rate fromexample 12, but they are statistically more in line with likely numbersproduced for further untested data for Second provider.

     Options:    Use 75% of data for training    Pruning confidencelevel 99%    Cross-validate using 10 folds  Class specified by attribute{grave over ( )}inStory’  Read 3873 cases (84 attributes) fromLOS_SECOND PROVIDER data  Read misclassification costs from LOS_SECONDPROVIDER costs data  [ Fold 1 ]  Decision tree excerpt:  eligible = f:out (282)  eligible = t:  :...APRDRG in49,55,321,317,912,950,98,563,196,565,205,811,813,110,510,193,  :   770,191,530: in (0)   APRDRG in774,301,753,203,776,198,206,252,207,315,190,773,309,   :   280: in(180/7)   APRDRG in468,812,383,640,540,751,221,710,421,500,756,302,757,463,171,140,  :   139,446,351,134,53,466,45,425,137,135,246,861,711,304,111,724,  :   422,349,263,229,42,130,283,54,663,362,754,420,340,240,57,58,531,  :   513,197,721,314,26,380,141,44,136,251,951,342,47,143,650,723,  :   364,226,305,82,46,114,566,228,225,755,347,465,142,313,223,385,  :   791,384,113,343,424,758,661,227,816,222,541,281,483,138,41,461,  :   341,501,722,892,441,346,681,561,43,815,144,48,690,952,344,50,  :   660,581,381,633,626,224,546,80,639,115,544,310,691,930,56,382,  :   121,200,220,752,442,519,262,482,760,443,51,512,445,  :   862: out(1799)     Evaluation on hold-out data (388 cases):    Decision Tree  -----------------------   Size  Errors Cost    65 11( 2.8%) 0.03 << [Summary ] Fold   Decision Tree ----  -----------------------  Size  Errors Cost   1  60  2.6% 0.05   2  66  0.8% 0.01  3  63  3.1% 0.05   4  66  1.3% 0.01   5  60  2.6% 0.05  6  66  3.1% 0.03   7  63  3.1% 0.05   8  66  2.3% 0.02  9  66  1.5% 0.06  10  65  2.8% 0.03  Mean 64.1  2.3% 0.04 SE  0.8  0.3% 0.01    (a) (b)  <−classified as   ---- ----   977  6 (a): class in    84 2806  (b): class out

Automatically-generating rules is a viable approach for synthesizing,encompassing, and deploying machine-learned “knowledge” that shows thata patient has or possibly will go “out of the norm” for the cost for amedical situation for various types of metrics, such as LOS. Rules,especially rules written in a textual way that conveys the logicalantecedent and consequent of those rules, can be very useful if humanunderstanding of the logic is required. This surface understanding,however, can quickly become a mirage of knowledge. Rules can give theimpression that a person can somehow understand the underlyingprocessing or process that is occurring.

Humans may understand the underlining process if the rules are small,the number of rules is not great, or the number of variables involved ineach rule is not considerable. As can be seen by the above examples,usually none of these conditions is true. Humans often get a false sensethat they can understand the rules when in fact the logic the rules ormodel convey is beyond our comprehension.

In some embodiments, deep learning neural networks may be trained toproduce a model to predict LOS or other metrics. If sufficientlytrained, neural networks can fire (determine “out of norm” or not) onminimal data. The above example decision trees comprised of rules willonly fire the consequent of a rule if the antecedent of that rule isentirely and precisely met. Neural networks can be trained to fire withless data while showing that the strength of that firing is less. Neuralnetworks, if trained with deep layers, can find nuance in thedata—relationships of data and outcomes—that rules may never demonstrateand that humans are incapable of finding.

Neural networks and especially deep neural networks are a black box. Onecan look at the weights of the connections of the neurons and never knowwhat any of it really means. With the above decision tree rules, a humancan look at, think about, and consider the ramification of the rules.Neural networks just feed data to and see what comes out. You will neverknow why it comes up with outputs—other than the training data possessedthe “shadow” of that relationship. Deep neural network processing ispractical because of the speed and ubiquity of parallel processors thatcan simultaneously perform large numbers of matrix multiplications.

The use of deep neural networks for predicting one or more metrics willallow recognition of greater nuance in the variables which point toin/out of norm. Groupings and relationships of the variables thatindicate in/out of norm may be detected with greater specificity.

Artificial intelligence (AI) is a field concerned with developingdecision-making systems to perform cognitive tasks that havetraditionally required a living actor, such as a person. Artificialneural networks (ANNs) are computational structures that are looselymodeled on biological neurons. Generally, ANNs encode information (e.g.,data or decision making) via weighted connections (e.g., synapses)between nodes (e.g., neurons). Modern ANNs are foundational to many AIapplications, such as automated perception (e.g., computer vision,speech recognition, contextual awareness, etc.), automated cognition(e.g., decision-making, logistics, routing, supply chain optimization,etc.), automated control (e.g., autonomous cars, drones, robots, etc.),among others.

Many ANNs are represented as matrices of weights that correspond to themodeled connections. ANNs operate by accepting data into a set of inputneurons that often have many outgoing connections to other neurons. Ateach traversal between neurons, the corresponding weight modifies theinput and is tested against a threshold at the destination neuron. Ifthe weighted value exceeds the threshold, the value is again weighted,or transformed through a nonlinear function, and transmitted to anotherneuron further down the ANN graph—if the threshold is not exceeded then,generally, the value is not transmitted to a down-graph neuron and thesynaptic connection remains inactive. The process of weighting andtesting continues until an output neuron is reached; the pattern andvalues of the output neurons constituting the result of the ANNprocessing.

The correct operation of most ANNs relies on correct weights. However,ANN designers do not generally know which weights will work for a givenapplication. Instead, a training process is used to arrive atappropriate weights. ANN designers typically choose a number of neuronlayers or specific connections between layers including circularconnection, but the ANN designer does not generally know which weightswill work for a given application. Instead, a training process generallyproceeds by selecting initial weights, which may be randomly selected.Training data is fed into the ANN and results are compared to anobjective function that provides an indication of error. The errorindication is a measure of how wrong the ANN's result was compared to anexpected result. This error is then used to correct the weights. Overmany iterations, the weights will collectively converge to encode theoperational data into the ANN. This process may be called anoptimization of the objective function (e.g., a cost or loss function),whereby the cost or loss is minimized.

A gradient descent technique is often used to perform the objectivefunction optimization. A gradient (e.g., partial derivative) is computedwith respect to layer parameters (e.g., aspects of the weight) toprovide a direction, and possibly a degree, of correction, but does notresult in a single correction to set the weight to a “correct” value.That is, via several iterations, the weight will move towards the“correct,” or operationally useful, value. In some implementations, theamount, or step size, of movement is fixed (e.g., the same fromiteration to iteration). Small step sizes tend to take a long time toconverge, whereas large step sizes may oscillate around the correctvalue, or exhibit other undesirable behavior. Variable step sizes may beattempted to provide faster convergence without the downsides of largestep sizes.

Backpropagation is a technique whereby training data is fed forwardthrough the ANN—here “forward” means that the data starts at the inputneurons and follows the directed graph of neuron connections until theoutput neurons are reached—and the objective function is appliedbackwards through the ANN to correct the synapse weights. At each stepin the backpropagation process, the result of the previous step is usedto correct a weight. Thus, the result of the output neuron correction isapplied to a neuron that connects to the output neuron, and so forthuntil the input neurons are reached. Backpropagation has become apopular technique to train a variety of ANNs.

FIG. 5 is a block diagram of an example of an environment including asystem for neural network training, according to an embodiment. Thesystem includes an ANN 505 that is trained using a processing node 510.The processing node 510 may be a CPU, GPU, field programmable gate array(FPGA), digital signal processor (DSP), application specific integratedcircuit (ASIC), or other processing circuitry. In an example, multipleprocessing nodes may be employed to train different layers of the ANN505, or even different nodes 507 within layers. Thus, a set ofprocessing nodes 510 is arranged to perform the training of the ANN 505.

The set of processing nodes 510 is arranged to receive a training set515 for the ANN 505. The ANN 505 comprises a set of nodes 507 arrangedin layers (illustrated as rows of nodes 507) and a set of inter-nodeweights 508 (e.g., parameters) between nodes in the set of nodes. In anexample, the training set 515 is a subset of a complete training set.Here, the subset may enable processing nodes with limited storageresources to participate in training the ANN 505.

The training data may include multiple numerical values representativeof a domain, such as red, green, and blue pixel values and intensityvalues for an image or pitch and volume values at discrete times forspeech recognition. Each value of the training, or input 517 to beclassified once ANN 505 is trained, is provided to a corresponding node507 in the first layer or input layer of ANN 505. The values propagatethrough the layers and are changed by the objective function.

As noted above, the set of processing nodes is arranged to train theneural network to create a trained neural network. Once trained, datainput into the ANN will produce valid classifications 520 (e.g., theinput data 517 will be assigned into categories), for example. Thetraining performed by the set of processing nodes 507 is iterative. Inan example, each iteration of the training the neural network isperformed independently between layers of the ANN 505. Thus, twodistinct layers may be processed in parallel by different members of theset of processing nodes. In an example, different layers of the ANN 505are trained on different hardware. The members of different members ofthe set of processing nodes may be located in different packages,housings, computers, cloud based resources, etc. In an example, eachiteration of the training is performed independently between nodes inthe set of nodes. This example is an additional parallelization wherebyindividual nodes 507 (e.g., neurons) are trained independently. In anexample, the nodes are trained on different hardware.

FIG. 6 is a block schematic diagram of a computer system 600 toimplement device and methods for implementing the database platform,including the expert system, and decision trees, and performing methodsand algorithms according to example embodiments. All components need notbe used in various embodiments. One example computing device in the formof a computer 600 may include a processing unit 602, memory 603,removable storage 610, and non-removable storage 612. Although theexample computing device is illustrated and described as computer 600,the computing device may be in different forms in different embodiments.For example, the computing device may instead be a smartphone, a tablet,smartwatch, smart storage device (SSD), or other computing deviceincluding the same or similar elements as illustrated and described withregard to FIG. 6. Devices, such as smartphones, tablets, andsmartwatches, are generally collectively referred to as mobile devicesor user equipment.

Although the various data storage elements are illustrated as part ofthe computer 600, the storage may also or alternatively includecloud-based storage accessible via a network, such as the Internet orserver-based storage. Note also that an SSD may include a processor onwhich the parser may be run, allowing transfer of parsed, filtered datathrough I/O channels between the SSD and main memory.

Memory 603 may include volatile memory 614 and non-volatile memory 608.Computer 600 may include—or have access to a computing environment thatincludes—a variety of computer-readable media, such as volatile memory614 and non-volatile memory 608, removable storage 610 and non-removablestorage 612. Computer storage includes random access memory (RAM), readonly memory (ROM), erasable programmable read-only memory (EPROM) orelectrically erasable programmable read-only memory (EEPROM), flashmemory or other memory technologies, compact disc read-only memory (CDROM), Digital Versatile Disks (DVD) or other optical disk storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other medium capable of storingcomputer-readable instructions.

Computer 600 may include or have access to a computing environment thatincludes input interface 606, output interface 604, and a communicationinterface 616. Output interface 604 may include a display device, suchas a touchscreen, that also may serve as an input device. The inputinterface 606 may include one or more of a touchscreen, touchpad, mouse,keyboard, camera, one or more device-specific buttons, one or moresensors integrated within or coupled via wired or wireless dataconnections to the computer 600, and other input devices. The computermay operate in a networked environment using a communication connectionto connect to one or more remote computers, such as database servers.The remote computer may include a personal computer (PC), server,router, network PC, a peer device or other common data flow networkswitch, or the like. The communication connection may include a LocalArea Network (LAN), a Wide Area Network (WAN), cellular, Wi-Fi,Bluetooth, or other networks. According to one embodiment, the variouscomponents of computer 600 are connected with a system bus 620.

Computer-readable instructions stored on a computer-readable medium areexecutable by the processing unit 602 of the computer 600, such as aprogram 618. The program 618 in some embodiments comprises software toimplement one or more of the database platform, expert system, and othermethods and operations described herein. A hard drive, CD-ROM, and RAMare some examples of articles including a non-transitorycomputer-readable medium such as a storage device. The termscomputer-readable medium and storage device do not include carrier wavesto the extent carrier waves are deemed too transitory. Storage can alsoinclude networked storage, such as a storage area network (SAN).Computer program 618 along with the workspace manager 622 may be used tocause processing unit 602 to perform one or more methods or algorithmsdescribed herein.

Although a few embodiments have been described in detail above, othermodifications are possible. For example, the logic flows depicted in thefigures do not require the particular order shown, or sequential order,to achieve desirable results. Other steps may be provided, or steps maybe eliminated, from the described flows, and other components may beadded to, or removed from, the described systems. Other embodiments maybe within the scope of the following claims.

The following statements are potential claims that may be converted toclaims in a future application. No modification of the followingstatements should be allowed to affect the interpretation of claimswhich may be drafted when this provisional application is converted intoa regular utility application.

1. A computer implemented method comprising: obtaining a schema definingvariables representing patient treatment experiences for one or morehealth conditions; obtaining data from historical patient treatmentexperiences, the data being arranged in accordance with the schema;training a machine learning system based on a training set of the datafor a specified time-period; generating a model from the trained machinelearning system, the model being based on variables selected by thetrained machine learning system to predict patient treatment experiencesfor the one or more health conditions; testing the trained machinelearning system on a set of test data from the obtained data fromhistorical patient treatment experiences; classifying test data patienttreatment experiences via the trained machine learning system;identifying the number of patient treatment experiences misclassified bythe machine learning system as “in story”; identifying the number ofpatient treatment experiences correctly classified by the machinelearning system as “out of story”; and generating a new model using oneor more qualifiers based on incorrectly classified in story patienttreatment decisions to increase the accuracy of the machine learningsystem.
 2. The method of claim 1, wherein the patient treatmentexperiences include at least one of a length of stay corresponding to apatient treatment procedure, readmittance following a hospital stay,outpatient visits turning into a hospitalization, emergency room visitsfrequency, complication rate, expenditure, population emergency roomvisit rate, population hospital admission rate.
 3. The method of claim2, wherein the data corresponds to a single health care facility andwherein the specified time-period comprises a time-period starting froma time following a change in procedures at the single health carefacility.
 4. The method of claim 3, and further comprising: generating afirst statistic for the patient treatment experience for a firsttime-period prior to the change in procedures; generating a secondstatistic for the patient treatment experience for a second time-periodfollowing the change in procedures; and determining the effectiveness ofthe change in procedures as a function of the patient treatmentexperience.
 5. The method of claim 4, wherein the length of staystatistic comprises a percent of patient treatment experiences that areabove a selected threshold over an average length of stay for arespective time-period.
 6. The method of claim 1, wherein the modelcomprises a decision tree including multiple rules and furthercomprising pruning the decision tree as a function of attribute usage inthe decision tree.
 7. The method of claim 1, wherein the model comprisesa decision tree including multiple rules and further comprising boostingthe decision tree by generating multiple different decision trees fromthe trained machine learning system and combining results from themultiple different decision trees to predict patient treatmentexperiences for the one or more health conditions.
 8. The method ofclaim 1, wherein the data is stored in a database in accordance with aschema defining the data comprising: a dependent variable name andcorresponding discrete value selected from a set of discrete values; afirst set of multiple independent variables having correspondingdiscrete values; and at least one independent variable having acorresponding continuous value.
 9. A computer implemented methodcomprising: accessing a trained machine learning system that has beentrained by: obtaining a schema defining variables representing patienttreatment experiences for one or more health conditions; obtaining datafrom historical patient treatment experiences, the data being arrangedin accordance with the schema; training a machine learning system basedon a training set of the data for a specified time-period; generating amodel from the trained machine learning system, the model being based onvariables selected by the trained machine learning system to predictpatient treatment experiences for the one or more health conditions;testing the trained machine learning system on a set of test data fromthe obtained data from historical patient treatment experiences;classifying test data patient treatment experiences via the trainedmachine learning system; identifying the number of patient treatmentexperiences misclassified by the machine learning system as “in story”;identifying the number of patient treatment experiences correctlyclassified by the machine learning system as “out of story”; generatinga new model using one or more qualifiers based on incorrectly classifiedin story patient treatment decisions to increase the accuracy of themachine learning system; and analyzing, using the trained machinelearning system, the patient discharge records to measure theperformance of a health care facility.
 10. The computer implementedmethod of claim 9, wherein measuring the performance of the health carefacility further includes identifying one or more aspects of theprovided health care that is above a baseline or identifying one or moreaspects of the provided health care that is below a baseline.
 11. Themethod of claim 9, wherein the patient treatment experiences include atleast one of a length of stay corresponding to a patient treatmentprocedure, readmittance following a hospital stay, outpatient visitsturning into a hospitalization, emergency room visits frequency,complication rate, expenditure, population emergency room visit rate,population hospital admission rate.
 12. The method of claim 11, whereinthe data corresponds to a single health care facility and wherein thespecified time-period comprises a time-period starting from a timefollowing a change in procedures at the single health care facility. 13.The method of claim 12 and further comprising: generating a firststatistic for the patient treatment experience for a first time-periodprior to the change in procedures; generating a second statistic for thepatient treatment experience for a second time-period following thechange in procedures; and determining the effectiveness of the change inprocedures as a function of the patient treatment experience.
 14. Themethod of claim 13, wherein the length of stay statistic comprises apercent of patient treatment experiences that are above a selectedthreshold over an average length of stay for a respective time-period.15. The method of claim 9, wherein the model comprises a decision treeincluding multiple rules and further comprising pruning the decisiontree as a function of attribute usage in the decision tree.
 16. Themethod of claim 9, wherein the model comprises a decision tree includingmultiple rules and further comprising boosting the decision tree bygenerating multiple different decision trees from the trained machinelearning system and combining results from the multiple differentdecision trees to predict patient treatment experiences for the one ormore health conditions.
 17. The method of claim 9, wherein the data isstored in a database in accordance with a schema defining the datacomprising: a dependent variable name and corresponding discrete valueselected from a set of discrete values; a first set of multipleindependent variables having corresponding discrete values; and at leastone independent variable having a corresponding continuous value.
 18. Asystem comprising: a processor and a memory device coupled to theprocessor and having instructions stored thereon for execution by theprocessor to perform operations comprising: accessing a trained machinelearning system that has been trained by: obtaining a schema definingvariables representing patient treatment experiences for one or morehealth conditions; obtaining data from historical patient treatmentexperiences, the data being arranged in accordance with the schema;training a machine learning system based on a training set of the datafor a specified time-period; generating a model from the trained machinelearning system, the model being based on variables selected by thetrained machine learning system to predict patient treatment experiencesfor the one or more health conditions; testing the trained machinelearning system on a set of test data from the obtained data fromhistorical patient treatment experiences; classifying test data patienttreatment experiences via the trained machine learning system;identifying the number of patient treatment experiences misclassified bythe machine learning system as “in story”; identifying the number ofpatient treatment experiences correctly classified by the machinelearning system as “out of story”; generating a new model using one ormore qualifiers based on incorrectly classified in story patienttreatment decisions to increase the accuracy of the machine learningsystem; and analyzing, using the trained machine learning system, thepatient discharge records to measure the performance of a health carefacility.
 19. The computer implemented method of claim 18, whereinmeasuring the performance of the health care facility further includesidentifying one or more aspects of the provided health care that isabove a baseline or identifying one or more aspects of the providedhealth care that is below a baseline.
 20. The method of claim 18,wherein the patient treatment experiences include at least one of alength of stay corresponding to a patient treatment procedure,readmittance following a hospital stay, outpatient visits turning into ahospitalization, emergency room visits frequency, complication rate,expenditure, population emergency room visit rate, population hospitaladmission rate.